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ABSTRACT 



This guide provides a "road map" to the various ways that 
schools and employers assess high-performance competencies, such as problem 
solving, information management, and communication and negotiation skills. 

The guide begins with a brief analysis of why it is important to assess these 
skills in light of the current standards environment in education and key 
trends in the economy. The guide includes an overview of tools and materials 
for assessing high-performance competencies and profiles of selected tools 
and materials. The methodologies range from "off-the-shelf" short answer 
tools to relatively expensive hands-on assessments. The profiles range from 
simples descriptions of existing materials to extended narratives of program 
or system development. Several of the strategies, especially those in the 
section on competency assessment in schools, outline processes that schools 
and school systems have developed in collaboration with partners. The tools 
are grouped into the following sections: (1) on-demand tests, both multiple 

choice and short answer; (2) self-assessments; (3) competency assessment in 
schools; (4) competency assessment in the workplace; and (5) competency 
assessment in adult education settings . This guide can be a starting point 
for those who want to equip young people with a broad range of cross-cutting 
high-performance skills to help them succeed in postsecondary education and 
the changing economy. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



TM032324 















LAELWorking Paper NonZL 



C ompetencies 

^ That Count 



u S DEPARTMENT OF EDUCATION 
Office of Educanonai Hesearcn ana imorovemem 

educational resources information 
S CENTER (ERIC1 

fl^This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have heen made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Strategies for assessing 
high-performance skills 

Lili Allen 



Northeast and Islands 
Regional Educational 
Laboratory. 





a program of 

The Education Alliance 
at Brown University 



222 Richmond Street, Suite 300 
Providence, Rl 02903-4226 



Phone: 401/274-9548 
Fax: 401/421-7650 
E-mail: info@lab.brown.edu 
Web: www.lab.brown.edu 





1 in partnership with 
Jobs For the Future 



T his guide provides a “road map” to the various ways that schools 
and employers assess high-performance competencies such as 
problem solving, information management, and communication 
and negotiation skills. The importance of such competencies is 
becoming increasingly evident because of their relevance both to 
school and to job performance. The guide begins with a brief 
analysis of why it is critical to assess these skills in light of both the 
current standards environment in education and key trends in the 
economy. The guide includes an overview of tools and mate rials for _ 
assessing high-performance competencies and profiles of selected 
cools and materials. 

The methodologies described here range from “off-the-shelf, short- 
answer tools to relatively expensive, hands-on assessments. While 
some assess high-performance competencies in a relatively tradi- 
tional manner, through multiple-choice questions about hypotheti- 
cal scenarios, others document and assess the ability of students to 
grapple with messy, real-world problems and to choose the appropri- 
ate tools and methodologies for accomplishing a task. The former 
assess knowledge of cross-cutting competencies and are likely to 
withstand challenges of reliability and validity, while the latter assess 
how students use cross-cutting competencies in practice and demon- 
strate what students can do in actual situations. 
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Because of the variability in methodologies, the 
profiles range from simple descriptions of existing 
materials to extended narratives of program or system 
development. Several of these strategies — especially 
most of those in the section “Competency Assessment 
in Schools” — oudine processes that schools and school 
systems have developed in collaboration with partners. 
The New Hampshire Competency-Based Transcript, 
for example, is the story of a forward-thinking state 
department of education that engaged with business 
and higher education partners to create a statewide 
system to capture student learning of high-perfor- 
mance competencies. Another state could adapt the 
resulting transcript, but, more importantly, New 
Hampshire’s system-building process could be used to 
develop a tool that reflects another states needs and 
strengths in educational and workforce development. 

Other strategies have been developed for a specific 
setting (e.g., community colleges, adult education 
centers) and could be adapted for alternative settings. 
For example, teachers seeking ways to assess student 
skill gains in the completion of projects could adapt 
the technique of behavioral interviewing, which is 
widely used in the corporate sector. We believe that 
the methodologies described here can promote cross- 
fertilization among the secondary education, adult 
education, corporate, and community college fields 
and inspire educators to try a range of strategies to 
teach and assess these high-performance competencies. 



Why Assess High-Performance 
Competencies 

As we turn the corner to the new millennium, more 
and more states and districts are implementing new 
high-stakes exit exams that hold both a promise and a 
threat for young people: these exams promise to hold 



all students to the same high standards, but, at the 
same time, they threaten to crowd everything except 
traditional academics out of the curriculum. 

This state of affairs is surprising given the fou nd a tion 
upon which the standards movement rests. Early in 
the standards movement, there was much talk about 
structural changes in the nature and organization of 
work, the labor market, and the global economy, and 
how these changes create a need for a new set of high- 
performance competencies. Educators, policymakers, 
industry leaders, and academics concurred: to succeed 
in todays economy, adults need cognitive, interactive, 
and problem-solving skills, along with more sophisti- 
cated academic skills than school systems were offer- 
ing. Most notably, a 1991 Labor Department blue- 
ribbon task force identified so-called SCANS skills . _ 

that it considered critical for economic success in the 
modern workplace. 1 According to the task force, 
these skills included cognitive skills, personal qualities, 
and competencies such as the abilities to collect and 
manage information, identify and use resources, work 
with others, and understand and design systems. At 
the same time, parents in upper-income communities 
became concerned that their children might not be 
gaining the critical thinking skills that are essential for 
professional careers, while literature emerging from 
graduate schools stressed the need for students who 
are completing advanced academic work to have such 
high-performance skills as teamwork and problem- 
solving. 

In recent years, however, the notion of a common core 
of cross-cutting skills has been all but forgotten in the 



1 SCANS stands for Secretary (of Labors) Commission on 
Achieving Necessary Skills. See U.S. Department of Labor, 
Secretary’s Commission on Achieving Necessary Skills, 
What Work Requires of Schools: A SCANS Report for 
America 2000 (Washington, DC: U.S. Department of 
Labor, 1991). 
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flurry of high-stakes, state- and district-level, stan- 
dardized testing. For reasons of simplicity, cost, and 
validity, most states have introduced standardized 
tests that reward the mastery of traditional subject 
matter and thereby drive teachers to focus their 
classes on that content. Because a students achieve- 
ment of a certain level of academic skill and knowl- 
' edge traditionally represents a gateway to college, 
most educators and the public also perceive these as 
the best predictors^of a persons ability to thrive in 
the economy and as a community member. 

Nevertheless, traditional academic skills and knowl- 
edge are not enough to help students address the 
ambiguous, complex, semi- or unstructured prob- 
lems of the real world. In fact, while often seen as 
“career competencies” of value only in the workplace, 

- high-performance skills are as important for success 
in K-12 and postsecondary education as they are in 
employment. Their centrality increases the higher 
one goes in education, and, as a result, they are 
becoming part of curricula and pedagogy in law 
schools, medical schools, and business schools. 

Instead of artificially separating career competencies 
from academic or intellectual competencies, we need 
to focus on preparing young people to succeed in all 
levels of formal education, in the workplace, and in 
community life. 

As any educator knows, the best way to encourage 
the teaching and learning of a more integrated set of 
skills is to assess those skills. However, many schools, 
districts, and communities choose not to try to 
incorporate large-scale performance assessments into 
their content standards and assessment systems, in 
part due to complications concerning validity. At the 
same time, and even though high-stakes, standard- 
ized tests are the most visible aspect of standards- 
based reform, some communities are choosing to use 
multiple measures to demonstrate student outcomes. 



They are adding the assessment of competency assess- 
ments — the assessment of high-performance or cross- 
cutting skills — to their portfolio of measurement tools. 

Competency assessments are attractive for a number of 
reasons: 

■ Competency assessments can provide a means to 
capture learning that takes place in workplace and 
community-based learning experiences. 

■ They can indicate the extent to which students 
who may not perform well on traditional stan- 
dardized tests have gained important knowledge 
and skills that will affect their future success. 

■ They can indicate ways to enrich curricula to 
include additional cross-functional skills. 

■ They can provide a bridge to local workforce 
development initiatives. 

Perhaps most importantly, high performance assess- 
ments can show us what students can do rather than 
focusing only on where their deficits lie. 



Tools and Methodologies for 
Assessing High-Performance 
Competencies 

We have cast a wide net in surveying and identifying 
methodologies for assessing cross-cutting competencies. 
While some originated in the high school building, 
others were designed to help corporations identify 
qualified entry-level employees or were created by 
collaborations among community-based organizations, 
schools, and business partners. This list is by no means 
exhaustive. Instead, it identifies types of assessments, 
presents one or two examples of each type, and, to the 
extent possible, illustrates how the assessments have 
been used. 
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For ease of navigation around the methodologies, the 
assessments are grouped here into five categories: 

■ On-demand tests , both multiple-choice and short - 
answer: Work Keys (ACT, Inc.); Workplace 
Success Skills (Learning Resources, Inc.) 

Self-assessments: Working (H &: H Publishing 
Company) 

■ Competency assessment in schools : New Standards 
Project (National Center on Education and the 
Economy and the University of Pittsburgh); New 
Hampshire’s Competency-Based Transcript; 
Coalition of Essential Schools/Transitions 
Project; SCANS 2000 Center/ AES International 

■ Competency assessment in the workplace: Massa- 
chusetts Work-Based Learning Plan; Behavioral 
Interviewing; SHL Job Simulations 

■ Competency assessment in adult education settings: 
Equipped for the Future 



How to Approach the Assessment 
Tools 

As you review the descriptions of tools and method- 
ologies, bear in mind the context in which you plan to 
implement this type of assessment. Adapting the tools 
and methodologies requires various levels of involve- 
ment of and preparation by the teacher and commu- 
nity partners. In general, to make it possible to assess 
cross-cutting competencies, educators and business/ 
community partners need to: 

1 Identify the specific skills that a project or com- 
munity experience , such as an internship , would 
be likely to develop . Most of the methodologies 
described here identify and define what skills 
they assess; schools might wish to use a tool that 
aligns with what their students have an opportu- 



nity to learn. Alternatively, some schools may 
want to create a process to identify, with their 
students or faculty, what skills they would like 
students to attain through projects, courses, or 
course sequences. For example, as a method of^ 
developing faculty support for reform, schools 
involved in the Coalition of Essential Schools 
engage faculty in determining what students 
should be able to know and do. Similarly, an 
individual teacher might work with a class to 
determine what skills should be gained through 
particular assignments or activities, or a commu- 
nity partnership might bring together employers 
and school partners to identify skills that work- 
based learning experiences should offer. 

2 Determine the primary purpose of the assessment ; 
It may be: 

■ To give feedback to the student for mid- 
course corrections as part of the learning 
process. The Massachusetts Work-Based 
Learning Plan has mid-course check-ins 
embedded in the course of documentation. 

■ To offer additional skills not gained elsewhere 
in the curriculum. For example, through an 
adaptation of the behavioral interviewing 
process, students could become more reflec- 
tive of their learning and learn to gather 
evidence of and articulate the skills they have 
gained. 

■ To provide a culminating evaluation. Tradi- 
tionally, assessments have been used for this 
purpose. 

■ To document, and perhaps accredit, skills so 
that students can convey their skills to 
postsecondary institutions or employers. Most 
of the “ready-to-use” evaluations are legally 
defensible. The New Hampshire competency- 
based transcript was designed collaboratively 
to be a portable record of learning for both 
postsecondary institutions and employers. 
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Tools and Methodologies for Assessing High Performance Skills 



Assessment 


Method 


Audience/ 

Uses 


Competencies 


Tool/method 

requires 

customization 


Level of system 
change required/ 
assumed 


Work Keys 


Paper-and-pencil; 
some video/audio 


Schools, regional 
partnerships, 
businesses, states 


Applied 
mathematics; 
applied technology; 
listening; locating 
information; 
observation; reading 
for information; 
teamwork; writing 


No 


None 


Workplace 
Success Skills 


Video-based paper- 
and-pencil 


Schools, regional 

partnerships, adult 

workforce 

development 

programs, 

businesses 


Interacting with 
others; listening; 
structuring work 
activities; 

trainability; graphs 
and charts 


No 


None 


Skill Command 


Rubric 


Schools, regional 

partnerships, 

businesses 


Teamwork; 
interpretation and 
use of information; 
problem-solving 


No 


None 


SHL Job 
Simulations 


Simulations 


Job interviews in 
private sector 


Functioning in 
group and team 
settings; organizing 
and prioritizing 
tasks; conducting 
research, analysis, 
and idea generation 
to solve problems 


With adaptation, 
could be used by 
schools 


None 


Working 


Paper-and-pencil 

self-assessment 


Schools, colleges, 
adult workforce 
development 
programs 


Taking responsibility; 
working in teams; 
persisting; sense of 
quality; lifelong 
learning; adapting 
to change; 
permanent 
problem-solving; 
information 
processing; systems 
thinking 


- No 


None 


New Standards 
Applied Learning 
Standards 


Work samples with 
rubrics 


Districts, for 
implementation in 
classrooms 


Problem solving; 
communication 
information; 
learning and self- 
management; 
working with others 


No. Some districts 
customize by 
developing locally 
based exemplary 
work samples 


Requires professional 
development to help 
teachers incorporate 


standards into 
teaching practice 
and to assess 
student performance 


New Hampshire's 
Competency- 
Based Transcript 
System 


Exemplars/ 
locally developed 
rubrics and 
portfolios 


Piloting in fourteen 
schools in New 
Hampshire; eventual 
use by all New 
Hampshire schools 


Decision making 
and problem-solving; 
self-management; 
communication; 
ability to work with 
others; information/ 
use of technology, 
research, analysis 


Yes. Districts in 
other states would 
have to customize 
because the 
transcript is based 
on New Hampshire's 
curriculum 
frameworks 


System is the result 
of collaboration 
between state DOE, 
post-secondary, and 
business partners; 
requires professional 
development for 
implementation 


Coalition of 
Essential Schools/ 
Transitions Project 


Rubrics/portfolios 
as part of 

alternative transcript 


Piloting in two 
schools in California 


Coalition of 
Essential Schools' 
Habits of Mind, 
e.g,, habits of 
inquiry; technology; 
collection and 
organization of 
information; 
communication of 
ideas; effective 
cooperation with 
others; lifelong 
learning 


Yes. Individual 
schools identify own 
learner outcomes 
and revise rubrics 
accordingly 


Requires agreement 
with postsecondary 
institutions to credit 
transcript for 
admissions; requires 
professional 
development on 
scoring for teachers 
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■ To provide an opportunity for a teacher, 
school, district, or community to determine 
whether the curriculum overall is delivering 
particular skills. Work Keys has been used in 
this regard in several localities. 

■ To give a diagnostic report to a community 
regarding the skills students are gaining 
through the existing curriculum. For example, 
Ohio used Work Keys to determine whether 
stu dents were prepared for the states fastest- 
growing jobs, then devised a strategy to 
address skill gaps. 

3 Determine what “evidence” of student learning 
and achievement the relevant stakeholders will 
require . Most often, these stakeholders are 
“gatekeepers” such as employers and institutions 
of higher education. The New Hampshire 
Department of Education and the California 
Transitions Project have focused on making 
credible the attainment of cross-cutting compe- 
tencies to postsecondary institutions and employ- 
ers. The SCANS 2000 Centers Career Transcript 
system allows users to post results from Work 
Keys and from Work Success Skills for a portable 
“passport” of learning. 

4 Determine hoiv much change your system — school 
department , school! community partnership, or 
school — will be willing to undertake. For ex- 
ample, implementation of the New Standards 
requires professional development to help 
teachers incorporate the standards into teaching 
practice. Several of the tools described here 
require the creation of “assessor networks” within 
or across schools so that teachers and other 
assessors can come to agreement on process and 
standards. In addition, many of the tools identi- 
fied here require adaptation by local users, as 
noted in matrix above. 



On-Demand Tests, Both Multiple- 
Choice and Short-Answer 

Several companies have developed “ready-to-use” 
assessment systems that businesses can use to assess job 
applicants or that schools or industry/education 
partnerships can use to assess student and worker skill 
levels. These methodologies have the advantage of 
being legally defensible. Also schools, community 
organizations, and school/community partnerships can 
adapt them to identify skill gaps and the training needs 
of students and potential workers. These assessment 
systems identify knowledge of competencies, rather 
chan the ability to apply competencies in context. 

WORK KEYS - 

Work Keys is a three-stage employability skills assess- 
ment tool that includes job analysis, assessment, and 
instructional support. Widely used by industry, educa- 
tion systems, and business/education consortiums, the 
Work Keys system is built around a set of scales that 
measure both the skills of individuals and the compe- 
tency levels required for successful job performance. 
Work Keys is a program of ACT, Inc., an organization 
most widely known for its college admissions testing 
service. 

Work Keys provides highly detailed information about 
skill levels in eight “foundation skills,” or “the skills 
needed to learn other skills:” 

■ Applied Mathematics: Applying mathematical 
reasoning to work-related problems; 

■ Applied Technology: Basic principles of mechanics, 
electricity, fluid dynamics, and thermodynamics as 
they apply to machines and equipment found in 
the workplace, with the emphasis on problem- 
solving; 
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■ Listening: Listening to and understanding work- 
related messages; 

■ Locating Information: Locating, comparing, 
summarizing, and making decisions using 
information in workplace graphics (e.g., dia- 
grams, floor plans, tables, forms, graphs, charts, 
instrument gauges); 

» i ~ 

a Observation: Noticing details and paying atten- 
tion to instructions and demonstrations; 

■ Reading for Information: Reading and under- 
standing work-related instructions and policies in 
memos, bulletins, notices, letters, policy manuals, 
and governmental regulations; 

■ Teamwork: Choosing behaviors and/or actions 
that simultaneously support relationships within 
the team and lead toward the accomplishment of 
work tasks; and 

■ Writing: Writing work-related messages and 
summaries. 

Each skill area is scaled from the lowest level for which 
employers are willing to test up to the point at which 
specialized training is needed. Several of the skills are 
assessed through traditional, paper and pencil testing, 
while others incorporate the use of audio or videotapes 
that provide prompts or scenarios to which the 
student must respond. Testing time ranges from 40 to 
80 minutes. 

Work Keys materials suggest that students work with 
counselors to take the assessments and pinpoint areas 
for improvement. The accompanying Targets for 
Instruction manuals are designed to be incorporated 
into classroom curricula; they include descriptions of 
the skill levels, examples of skill building strategies, 
and work-related scenarios at each skill level. 

In Ohio, the Ohio Business Roundtable and the Ohio 
Department of Education hired ^CT to both profile 



the states fast-growing jobs and assess the skill levels 
of 14,000 high school seniors from urban, suburban, 
and rural communities. The results indicated that only 
one in fourteen seniors met the profile across four skill 
areas (applied mathematics, reading for information, 
applied technology, and locating information). The 
initiatives recommendations included raising aca- 
demic standards, increasing opportunities for work- 
based learning, and continuing the use of skill-based 
assessments. 

In California, the Tri-Valley Education Collaborative 
has used Work Keys as a diagnostic tool to pinpoint 
needed changes in curricula and, less extensively, with 
a small group of students to assist them in undertak- 
ing more careful career planning. In 1998, the Col- 
laborative piloted the Work Keys assessment with 
about 500 students from comprehensive and alterna- 
tive schools, then gathered district leaders to discuss 
aggregated results for their schools. The Collaborative 
hopes to build on that pilot by engaging schools in 
extended discussions about what curricular changes 
would be necessary to improve student scores. 



Work Keys, ACT Inc, 3355 Lenox Road, NE, Suite 320, 
Atlanta, GA 30326-1332 (404) 231-1952; 
www.act. orglworkkeysl tour! index, html 



WORKPLACE SUCCESS SKILLS 

Learning Resources, Inc., provides a video-based, 
criterion-referenced SCANS assessment used in high 
schools, businesses, postsecondary institutions, and 
welfare training programs. The Workplace Success 
Skills (WSS) assessment system was developed and 
validated in collaboration with industry representa- 
tives. Educators use it to assess curricula and to assist 
students in identifying strengths and weaknesses; 
employers use it to screen applicants for entry-level 
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jobs and to give feedback to employees so that they 
may upgrade their work-relevant skills. WSS is also 
being used by the SCANS 2000 Center, described 
below, in three ways: (1) as a pre- and post-test to 
assess community-college students' acquisition of 
SCANS skills; (2) to assess the basic skills of partici- 
pants in welfare-to-work programs for the purpose of 
providing developmental feedback; and (3) as part of 
an incumbent worker/organized labor program funded 
by the U.S. Department of Labor. 

WSS assesses thinking skills (creative thinking, deci- 
sion making, problem solving, knowing how to learn, 
and reasoning) and personal qualities (responsibility, 
sociability, self-management, and commitment to 
quality). 

There are five sections to the assessment: 

■ Interacting with Others: The participant views 
brief vignettes of work situations with hourly 
workers and their supervisors as central charac- 
ters. Periodically, the video stops; the participant 
must respond to multiple-choice questions 
requiring judgment about actions or decisions. 

■ Listening: The participant views video vignettes 
that present information and instructions orally; 
they must answer multiple-choice questions 
about factual content interpretation. 

■ Structuring Work Activities: A supervisor gives the 
participant oral instructions on a complicated 
task; the participant must chronologically list the 
actions required to complete the task. 

■ Trainability: The participant answers multiple- 
choice questions on a training video covering 
complex material. 

■ Graphs and charts: The participant answers 
multiple-choice questions about information 
contained in graphs and charts. 



If all five modules are used, the assessment takes about 
2.5 hours and requires a PC with either Internet access 
or a modem, a VCR with monitor, and paper and a 
pencil. The participant views the video scenarios and 
puts answers to questions onto an answer sheet. A test - 
administrator then transfers the answers to the PC, * 
which sends them to a host computer for scoring. 
Finally, the host computer transmits a comprehensive 
feedback report back to the local computer. Because 
the host computer does the scoring, no professional 
development costs are associated with WSS. As of 
September 2000, a modem will no longer be required: 
data and reports may all be scored by the Internet. 

The feedback report contains: 

■ Probability-of-success information, based on a 
comparison of the examinees scores against a 
validation group (the database of scores exceeds 
one million people); 

■ A rank ordering of the participant's facilitation 
skills, influence skills (guiding the actions and 
thinking of others), customer service orientation, 
problem-solving skills, and listening skills; 

■ A detailed report showing areas where the 
participants performance was satisfactory and 
where development is warranted; and 

■ Suggested activities for improving performance 
in the assessed skill areas. 

In addition to the feedback report, resources designed 
to support educators and trainers include a facilitators 
guide to manage discussion questions about the skills 
measured by WSS. A comprehensive list of print- 
based scenarios has been designed to foster discussion 
and the development of individuals' competence in 
areas measured by the program. 

In Indiana, Ivy Tech State College has worked with 
the state's Department of Workforce Development 

s 
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and a consortium of businesses in a particular com- 
munity to assess employees’ skill levels and to identify 
areas of opportunity for collaborative training. The 
WSS feedback reports provided employees with 
feedback targeting opportunities for individual’s 
development. Group reports provided individual 
organizations with information regarding their 
workforce as a whole and where training was required 
to bring groups to higher levels of performance. 
Finally, reports-provided information regarding 
potential areas for collaboration in investing training 
and development dollars. 



Workplace Success Skills, Learning Resources Inc., 700 
Canal Street, Stamford, CT 06902-5921 (203)637- 
5047; www. learning-resources, com 



Self-Assessments 

WORKING: ASSESSING SKILLS, HABITS, AND STYLE 

“Working” is a statistically valid self-assessment tool 
that can be used in a wide range of educational and 
training settings to assess nine SCANS competencies: 
taking responsibility, working in teams, persisting, a 
sense of quality, lifelong learning, adapting to change, 
permanent problem-solving, information processing, 
and systems thinking. 

The assessment is administered in 30 minutes and can 
be self-scored. Individuals complete 50 questions, 
such as “I usually do something I enjoy rather than try 
something different” or “When learning something, I 
first think carefully about the very best way to tackle 
it.” For each question, individuals rate the statements 
as “Almost never like me; Occasionally like me; 
Moderately like me; Quite a bit like me; or Almost 
always like me.” ^Working includes materials that can 



be used in worksites or training programs as follow-up 
activities to help individuals determine ways to 
strengthen their behavior in particular areas. 

The Lucent Technologies Manufacturing Workforce 
Collaborative used Working to assess student learning 
in a curriculum pilot project that targeted five general 
skill areas identified as necessary for the manufacturing 
workforce: personal, interpersonal, systems, proce- 
dures, and information. Fifty-seven incumbent work- 
-ers participated in the curriculum pilot project, 
completing the Working self-assessment as a pre- and 
post-test. While the population size was too small to 
generate statistically valid findings on gains/losses in 
any of Working’s nine competencies, the Collaborative 
cited percentile increases in “Responsibility and 
“Teamwork” as indicative of the impact of the training. 



Working: Assessing Skills, Habits, and Style, H&H 
Publishing Company, Inc., 1231 Kapp Drive, 
Clearwater, FL 33765 (800) 366-4079; 

www. hhpublishingcorn 



Competency Assessment in — — 
Schools 

NEW STANDARDS PROJECT'S APPLIED LEARNING 
STANDARDS 

The New Standards Project, a partnership between the 
National Center on Education and the Economy and 
the University of Pittsburgh, has designed Applied 
Learning Standards. Part of a package of performance 
standards that includes English Language Arts, Math- 
ematics, and Science, the Applied Learning Standards 
are one of the most established systems for assessing 
cross-cutting competencies in an academic context. 
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The New Standards Project does not propose that 
Applied Learning be a separate area of study but that 
students draw from other subjects to meet the stan- 
dards. The project elected to develop separate stan- 
dards, however, to ensure that cross-cutting competen 
cies do not get lost in the demands of more academic 
subjects. The five performance standards are: 

■ Problem solving; 

■ Communication tools and techniques; 

■ Information tools and techniques; 

■ Learning and self-management tools and tech- 
niques; and 

■ Tools and techniques for working with others. 

The standards are set at the level of the tenth grade, 
consistent with the partners’ interest in all students 
across the country completing a common academic 
core by the end of the tenth grade and achieving a 
Certificate of Initial Mastery. 

The New Standards Project describes the problem- 
solving standard as the “centerpiece” of the standards. 
It defines projects organized around three kinds of 
problem solving: 

■ Design a product, service, or system in which the 
student identifies needs that could be met by 
new products, services, or systems and creates 
solutions for meeting them; 

■ Improve a system in which the student develops 
an understanding of the way systems of people, 
machines, and processes work, troubleshoots 
problems in their operation, and devises strate- 
gies for improving their effectiveness; and 

■ Plan and organize an event or an activity in 
which the student takes responsibility for all 
aspects of planning and organizing an event or an 
activity from concept to completion . 2 



The four other standards describe tools and tech- 
niques that can be used to complete projects that meet 
the problem-solving standards. The New Standards 
partners are clear that these tools and techniques 
should not be learned and practiced in a piecemeal 
way; they must be integrated into projects, and 
projects should involve subject matter related to the 
core subject standards. 

As with its core subject standards, the New Standards 
Project defines each standard and delineates perfor- 
mance descriptions that explicitly describe what 
students should know and be able to do. The perfor- 
mance descriptions are concrete and defined in terms 
of actual products, services, or performances that 
students can use to show evidence of achievement of 
the standard. 

The project includes samples of standard-setting 
student work, collected from more traditional “aca- 
demic disciplines, as evidence of proficiency in applied 
learning standards, and it includes commentary on 
how the work meets the standards. The work samples 
and commentary are detailed, including a description 
of the task, the circumstances of performance, thef 
student work, and an evaluation, of. what the sample-of, 
student work shows regarding student performance of 
the standards. Teachers can use the criteria described 
in the Applied Learning Standards to benchmark their 
own students’ work. The projects cross-referencing 
system enables teachers and students to identify ways 
in which projects may be used to show evidence of 
meeting more than one standard in Applied Learning 
as well as in academic disciplines. 

The Fort Worth (Texas) Independent School District 
collaborated with the New Standards Project by 
serving as an implementation site for the pilot New 

2 New Standards Performance Standards, Volume 3: High 
School, p. 106. 
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Standards in the early 1990s. The district had under- 
taken a survey of employers and determined that 
many of the skills required in the Fort Worth 
economy were not being taught in the schools, and it 
had initiated a project to incorporate workplace 
competencies into academics. 

Every summer, a new cohort of 30-40 teachers is 
trained in applied learning pedagogies: teachers 
develop projects in their content area (i.e., math, 
science, or English) that incorporate workplace skills. 
The school system has done some work developing 
rubrics that map to the Applied Learning standards, 
based on adult models of performance. In addition, 
teachers participating in the summer training are 
encouraged to engage their students in developing 
rubrics tailored to each specific project and that take 
into account both content and the Applied Learning 
standards. Teachers and students work together to 
determine “how good is good enough? Students 
develop their sophistication in identifying, and 
producing, quality work through participating in the 
rubric development process. 

New Standards Project, National Center on Education 
and the Economy, 700 Eleventh Street, NW, Suite 750, 
Washington , DC 20001 (202) 783-3668; http://ncee,org 



NEW HAMPSHIRE'S COMPETENCY-BASED 
TRANSCRIPT 

New Hampshire is a leader, among New England 
states and nationally, in designing and piloting a 
competency- based transcript. A competency-based 
transcript is a tool for documenting and assessing 
skills and knowledge gained through both academic 
and out-of-school learning opportunities. It can 
include traditional measures of academic achievement, 

U 
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such as SAT scores and grades, as well as more innova- 
tive measures of student performance, such as judg- 
ments of proficiency. 

Under the leadership of the New Hampshire Depart- 
ment of Education, the transcript was designed through 
a collaborative process including representatives from 
secondary schools, post-secondary institutions, business 
and the community, in accordance with the state’s 
curriculum frameworks for career development, English 
Language Arts, Mathematics, Science, and Social 
Studies. Work groups identified a number of important 
features that would characterize New Hampshire’s 
transcript. These features included: combining tradi- 
tional reporting measures (grades) and newer compe- 
tency-based measures of proficiency; aligning the 
transcript with the state’s academic standards and career- 
development frameworks; and recognizing a variety of 
community and workplace learning experiences. 

The state Department of Education issued a Request for 
Proposals, asking school districts to apply to be Bench- 
mark Communities,” based on their ability to document 
evidence of progress in implementing education reform 
strategies and their interest in piloting the competency- 

based transcript. In the spring of 1998, the department 

designated four school districts as Benchmark Communities. 

In the winter of 1998-99, in preparation to use the 
competency-based transcript in the following school 
year, the Benchmark Communities refined the pilot 
document. They decided to modify the draft transcript 
to use summative, rather than course-based, judgments. 
In other words, the transcript would reflect a single 
judgment of student proficiency for each designated 
competency area at the end of a school year, drawing on 
evidence from various learning experiences. The pilot 
would focus only on documenting student abilities 
related to five high-performance skill sets or competen- 
cies: decision-making and problem-solving skills, self- 
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management, communication skills, ability to work 
with others, and information use (technology, re- 
search, analysis). Competencies in core academic areas 
would be incorporated in later years. 

Work groups, drawn from each of the Benchmark 
Communities, took on the task of developing com- 
mon guidelines for assessing and documenting student 
learning in relation to each of the high-performance 
skills. During the spring of 1999, working with Jobs 
for the Future and a consultant, the groups created a 
work plan for the pilot. Each of the four schools made 
a commitment to: 

■ Pilot the transcript in at least the tenth and 
eleventh grades; 

■ Include at least 1 5 percent of the students in each 
grade; 

■ Select a diverse, representative sample of students 
and faculty to participate; 

■ Offer students a variety of opportunities to 
collect evidence both within classes and from 
non-classroom settings; and 

■ Prepare all teachers at the pilot schools for 
mapping their courses against the transcript to 
show how and where students could collect 
evidence of mastery of the five high-performance 
skills. 

In September 1999, the four Benchmark Communi- 
ties began using New Hampshire’s pilot competency- 
based transcript, with a goal of eventual expansion to 
include the academic frameworks and transcript use 
for post-secondary admissions and employers. Each 
participating student became responsible for collecting 
his or her own work samples as evidence of mastering 
a particular competency, and then presenting this 
portfolio to a mentor/advisor or a designated assessor. 
Students submitted evidence of proficiency in the 



high-performance skills from academic courses and 
out-of-school learning experiences; they were judged 
as not meeting, meeting, or surpassing the standards. 
Teachers learned to assess portfolio collections of 
student work against the written performance-stan- 
dard for each competency, while students became 
acquainted with the performance standards and were 
coached in collecting evidence of proficiency for their 
transcripts. 

During the 1999-2000 school year, all the high 
schools used the same standards and the same format 
for the competency-based transcript. Each school 
developed a plan for internal processes to ensure 
consistent judgments among its assessors. Eventually, 
these judgments will be determined at each high 
school grade level and at exit from high school. As die^ 
pilot evolves, districts will work together to ensure 
that judgments of proficiency are determined consis- 
tently across schools; this will be key to ensuring that 
the transcript provides meaningful evidence of student 
learning statewide. 

Ultimately, New Hampshire’s objective is ambitious: 
to engage public and private post-secondary institu- 
tions in using the competency-based transcript. in the_ 
admissions process, and to involve employers in using 
the transcript as part of their hiring process. As one 
step toward that end, for the 2000-200 1 school year, 
the transcript pilot has expanded, and added ten new 
districts to the original four. In addition, the state has 
expanded the scope of the pilot transcript to assess 
student performance against proficiency standards in 
four core academic areas, oudined in the state’s 
curriculum frameworks, in addition to assessing 
student high performance skills. New Hampshire’s 
intention over the next few years is to use the pilot 
and other methods to expand interest in, and use of, a 
revised competency- based transcript statewide. 

13 
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Contact Mariane Gfroerer ; Guidance and Counseling 
Consultant , New Hampshire Department of Education 
78 Regional Drive y Concord \ NH 03301 (603) 271- 
6691. 



THE COALITION OF ESSENTIAL SCHOOLS/ 
TRANSITIONS PROJECT 

Schools that are members of the Coalition of Essential 
Schools network subscribe to ten principles of reform, 
one of which refers to methods of assessment: “Teach- 
ing and learning should be documented and assessed 
with tools based on student performance of real tasks. 
Multiple forms of evidence, ranging from ongoing 
observation of the learner to completion of specific 
projects, should be used to better understand the 
learners strengths and needs, and to plan for further 
assistance... The emphasis is on the students’ demon- 
stration that they can do important things.” 

In the process of determining what the “important 
things” are that students need to know and be able to 
do, many Essential schools have articulated a set of 
learner outcomes that encompass and go beyond 
traditional academic content mastery. For example, 
Hoover High School in San Diego expects all students 
to demonstrate progress on six learner outcomes that 
are similar to the SCANS skills: habits of inquiry, the 
experience of technology, collection and organization 
of information, communication of ideas, effective 
cooperation with others, and lifelong learning. 

Hoover students learn these skills as they move 
through ninth/tenth-grade interdisciplinary courses 
and eleventh/twelfth-grade “institutes” (or career 
pathways). Career exploration and community 
learning activities are a feature of all four years. 

A school-wide rubric that contains performance 
descriptors for each learning outcome is used to assess 

& 



student work, with a six-level scale ranging from 
“Distinguished” to “No Evidence.” “Achieves Stan- 
dard” indicates that the student has performed to 
grade level; “Mastery” and “Distinguished” indicate 
that the student has surpassed grade level. 

For the past five years, Hoover and another Coalition 
school, Homestead High School in Cupertino, have 
participated in the Transitions Project, a California- 
based initiative to document student learning through 
performance assessments for entrance into the state’s 
postsecondary institutions.^ Hoovers Transitions 
transcript documents a students performance in each 
subject area in relation to the learner outcomes. For 
example, a student might score “Mastery” in the 
learner outcome “Communication” in thejiumanities 
course, while scoring “Achieves Standard” in the 
learner outcome “Collaboration” in biology. While 
individual teachers score portfolio projects, Hoover 
used a consensus-scoring process to review student 
work for the more high-stakes transcripts required by 
the Transitions Project. To prepare for this, teachers 
participated in school-wide staff development on the 
consensus scoring of student work. 

At Homestead High School, three of the schools 
“Expected School-Wide Learning Results” — commu- 
nication skills, work skills, and thinking skills — cut 
across disciplinary boundaries and are developed as 
graduation-level, rather than grade-level, performance 
expectations. Assessments of these outcomes are 
designed to reflect students’ progress in moving from 



3 Several other high schools participated in the project, a 
collaboration among the California Center for School 
Restructuring, Pacific Education Group, and the Bay Area 
Coalition of Essential Schools. Only Hoover and Home- 
stead and one additional San Diego high school. Rancho 
Buena Vista, developed rubrics and participated in the 
process of submitting alternative transcripts to the Califor- 
nia public university systems throughout the life of the 
project. 
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“Novice” to “Exceeds Standards” in relation to a given 
standard. For example, while a ninth-grade student 
might be expected to score at the “Novice” level, s/he 
would be expected to progress to “Proficient” by 
graduation. The fourth Learning Result relates to 
standards that are specific to course content, and these 
are assessed separately according to grade-level expec- 
tations using a four-level scale, from “Novice” to 
“Exceeds Standards.” 

Homestead High School created, but later suspended', 
integrated studies programs in grades 9/10 and 11/12, 
organizing them around essential questions or prob- 
lems. Because the integrated studies programs were 
not school-wide, they involved only a subset of 
teachers and students in submitting alternative 
transcripts under the Transitions Project. However, 
other teachers, across the school, use the Transitions 
Project rubric. Because not all teachers use the perfor- 
mance assessments, Homesteads transcript includes 
both performance assessments, where available, and 
course grades. 

For more information, see "New College Admissions 
Procedures: Implications for Career-Related Learning in 
High School, ” National Center for Research in Vocational 
Education, August 1999; or go to the Coalition of 
Essential Schools Web Site: www.essentialschools.org. 



SCANS 2000 CENTER/AES INTERNATIONAL "SKILL 
COMMAND" 

SCANS 2000 at The Johns Hopkins University and 
collaborators have developed CD-ROM case studies 
designed to help community college students master 
SCANS skills within their academic coursework. The 
case studies simulate real workplace problems: they are 
“messy” situations requiring sustained effort, complex 

i 



decision making, research, and the use of knowledge 
and applications from many disciplines. Project 
developers identified problems, wrote the stories, 
included data (and gaps in knowledge), developed 
characters and dialogue, and developed the CD- 
ROMs to include simulated offices. 

Students work in teams to solve the problems, each of 
which addresses one or two primary SCANS skills as 
well as specific academic content. The CD-ROM 
provides enough data to get started and carefully 
placed tutorials on such topics as linear programming, 
statistics, and simulated flow-charting. Periodically, 
students are directed to stop the CD and complete 
activities. Students may arrive at solutions contained 
within the CD-ROM, or they may develop alternative 
solutions not envisioned by the project developers. 

To assess student learning, SCANS 2000 surveyed the 
field to identify tools for assessing some of the “hard 
to measure” skills, such as teamwork, that the CD- 
ROMs emphasize. The effort has selected two mea- 
sures, one for diagnostic purposes and one for stan- 
dardized assessment, and opted to pilot these in three 
SCANS skills: teamwork, interpretation and use of 
information, and problem-solving. ... ... 

For diagnostic assessment, SCANS 2000 turned to 
AES International, a company founded in 1993 for 
the purpose of providing tools in support of the 
recently released SCANS skills. SCANS 2000 worked 
collaboratively with teachers and AES International to 
develop assessment rubrics from AES’ “Skill Com- 
mand,” an assessment instrument in which various 
people (e.g, teachers, peers, employers, self) rate 
participants’ behaviors. The assessment rubric breaks 
each of the major skills into multiple behaviors on 
which teachers rate students on a scale from “No 
Understanding” to “Expert.” For example, the rubric 
lists seven behaviors under problem-solving, ranging 

15 
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from “Clearly Defines the Problem” and “Considers 
Alternative Solutions” to “Monitors Progress of 
Proposed Solution.” The instrument also specifies 
when to measure students on each of the skills. For 
example, students are assessed on problem-solving at 
the same time they are assessed on solving the 
advertising budget, using the CD-ROM, in an 
algebra class. The “teamwork” skill assessment is 
anchored to the task through which team members 
must select a car to market, and it assesses students’ 
cooperation with and contribution to their team. 
Each task is rated with a SCANS level of difficulty 
because, for example, problem solving in one venue 
may be more complex than in others. 

For a more objective pre- and post- assessment, the 
team selected the Work Success Skills video simula- 
tions, profiled above. It selected the WSS system for 
three reasons: the WSS videos measure the skills 
emphasized in the CD-ROMs; it is objective and 
industry-validated, predicting job success four times 
more accurately than an interview; and it generates 
data on student performance that includes informa- 
tion on strategies a teacher or the learner can use to 
address specific skills gaps. 

The SCANS 2000 team is piloting the assessment 
process with community college partners and will 
work with the faculty to expand the number of tasks 
and skills they will assess. Meanwhile, the SCANS 
2000 team, in partnership with AlignMark, has 
developed a Career Transcript System (CTS) that is 
being piloted throughout the country in high 
schools, universities, and social-service agencies. The 
CTS tracks and documents the attainment of 
SCANS skills. The participant takes part in an 
assessment such as Work Success Skills or Skill 
Command; the assessment results are posted to the 
CTS; and if the participant likes the data then s/he 



“authorizes” the information to be posted to the 
transcript, where employers can view it. The result is a 
transcript, similar to a resume in that it shows past 
job-relevant experience, yet different from a resume in 
that the employer is assured that the information is 
validated through an objective source. These tran- 
scripts are expected to facilitate entry into the 
workforce and provide a platform for documenting 
developmental change throughout a participant’s 
career. 



For more information about Skill Command, go to 
www.skillcommand.com. To learn more about SCANS 
2000, go to www.scans.jhu.edu. 



Competency Assessment in the 
Workplace 

The Massachusetts Work-Based Learning Plan was 
developed for the use of employers offering work 
experiences to high school students. It is a tool for 
assessing student learning of cross-cutting competen- 
cies in the workplace. The other two assessment 
methods described below — SHL Assessment Centers 
and Behavioral Interviewing — are used in the work- 
place to identify and develop entry-level and execu- 
tive-level employees; these have not yet been imple- 
mented in schools. Both are included here because 
they focus on training assessors to use observational 
skills to assess performance. School systems might 
adapt this methodology to assist teachers in using 
observational skills to identify ways in which students 
are, or are not, gaining high-performance skills in 
accomplishing school-related tasks. 
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MASSACHUSETTS WORK-BASED LEARNING PLAN 

In 1996, the Boston Private Industry Council (PIC) 
created a Work-Based Learning Plan to enhance the 
quality of work-based learning in its programs that 
link academic and career-related classes with work- 
based learning. The PIC revised occupation-specific 
training plans to focus instead on broad competencies 
that Boston employers, schools, and community 
members agreed were essential to success in higher 
education and the world of work. The learning plan is 
a master form that students, teachers, worksite super- 
visors, and PIC staff complete for each placement. It 
focuses students, employers, and school-based person- 
nel on a mutually agreed-upon set of skills and 
abilities that students should have the opportunity to 
develop. It also provides an opportunity for teachers 
and workplace supervisors to engage together in 
developing the skills and competencies of individual 
students, allowing for a greater connection between 
classroom and workplace learning. 

The Massachusetts Office of School-to-Career, 
through a statewide Work-Based Learning Statewide 
Steering Committee, has adopted the Work-Based 
Learning Plan for use across the state, and has created 
an electronic version of the Plan along with a guide 
for supervisors on its use. The Massachusetts Office of 
School-to-Career has also released a toolkit to help 
trainers across the state train employers, educators, 
students, and parents on how to ensure an effective 
work-based learning experience. The toolkit contains a 
CD-ROM with reproducible materials from the 
toolkit, including the Work-Based Learning Plan. 

The learning plan covers nine competencies of three 
overall types: 

■ Individual Competencies : communication and 
literacy; organizing and analyzing information; 
problem solving; using technology; and complet- 
ing entire activities; 



a Team Competencies: acting professionally; 
interacting with others; understanding all 
aspects of the industry; and 

b Personal and Professional Development Compe- 
tency: taking responsibility for career and life 
choices. 

Each of the nine competencies is further divided into 
more specific skills. The Work-Based Learning 
Statewide Steering Committee developed a rubric 
that specifies ratings for each skill, ranging from 
“Needs Development” to “Advanced.” For example, 
under the speaking skill (communication and literacy 
competency), a student rated “Needs Development” 
is “learning to speak clearly, audibly, and courte- 
ously;” a student rated “Proficient” “expresses com- 
plex ideas in a well thought-out, organized, and 
concise manner.” Under the research and analysis skill 
(organizing and analyzing information competency), 
a student who “Needs Development” is “developing 
familiarity with pertinent information;” an “Ad- 
vanced” student “develops theories of action and tests 
them in practice.” 

The learning plan asks supervisors to undertake a 
three-step process: - - 

1 Use the plan to identify the job description and 
tasks to be completed, then pick the three to five 
competencies most critical to the job. 

2 After observing the students on-the-job perfor- 
mance for a week or two, conduct an evaluation 
with the student and set goals in competency 
areas. 

3 At the end of the work experience, or at appro- 
priate intervals during longer placements, 
conduct a follow-up assessment meeting with 
the student. 

17 
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Instructions for the learning plan encourage supervi- 
sors to complete the ratings with the student so that 
students develop an awareness of their own skills levels 
and an understanding of the learning potential of the 
workplace. 

The PIC has adapted the learning plan for alternative 
education initiatives, such as Diploma Plus, a program 
for at-risk and out-of-school student participants that 
requires at least 150 hours of paid or unpaid work or 
community service. It has also introduced the learning 
plan and its competencies to job shadowing experi- 
ences: employers can use the competencies to show 
students the skills needed for the workplace, while 
teachers can use them in the classroom as a “checklist” 
that gives students an analytic framework to reflect 
upon their brief experiences and compare them to past 
or future shadows. 



Massachusetts Work-Based Learning Plan , Massachusetts 
Department of Education, 350 Main Street , Malden , 

MA 02148 (781) 388-3382; www.doe.mass.edu/stw/ 
mwblp.html. This web page has many tools and materials 
on the implementation of the work-based learning plan . 



BEHAVIORAL INTERVIEWING 

In behavioral interviewing, a strategy widely used by 
businesses, interviewers elicit actual examples of past 
behavior in order to predict future performance. Most 
often, the skills for which behavioral interviewers 
probe are “soft” skills, such as attention to detail, 
leadership, initiative, and decision making. As one 
private-sector user of this methodology put it, “Tech- 
nology skills are easy to measure. We use behavioral 
interviewing to help us identify which applicants have 
the leadership and organizational capabilities to 
succeed on the job.” Its proponents argue that ques- 




tions about what an applicant did are better predictors 
of behavior than questions about what an applicant 
would do . Behavioral interviewing focuses less on 
content or technical knowledge and more on how an 

applicant accomplished activities in previous employ- 

ment. 

Often, businesses use consultants to help them 
identify critical competencies for particular jobs and 
to train their staff on behavioral interviewing tech- 
niques. In this way, it is also a more efficient means of 
interviewing than most: interviewers can assess for 
particular skills, rather than use a broader assessment 
instrument to assess all cross-cutting competencies. 

Typically, a business will identify the competencies 
demonstrated in effective performance of a job, then 
develop job-related questions to gather examples of 
related past activities. For example, to probe for an 
applicant s attention to detail, an interviewer might 
ask the applicant to give an example of when s/he had 
to deliver a finished product under a tight deadline. 
The trained interviewer listens for clarification, paying 
attention to what descriptive words the applicant uses 
to describe the event. For example, the interviewer 
-might say: 

■ What were you thinking? 

■ What were you feeling? 

■ What was your contribution to that? 

■ Walk me through your thought process. 

■ Take me through that step by step. 

■ How did you know what to do? 

Afterward, the interviewer codes the interview, often 
using a rubric developed specifically for the identified 
competencies. 
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The Bank of America has adapted the behavioral 
interviewing process with its welfare-to-work program. 
In recognition of former welfare recipients’ previous 
experience, the bank’s human resources staff are 
trained to elicit examples of past performance in the 
home as well as in the workforce, and partnering 
community-based organizations train applicants who 
have been on welfare to develop examples from both 
work and home experience to identify competencies. 

The behavioral interviewing process contains clues as 
to how teachers might enhance their assessment of 
individual or group projects. For example, a teacher 
could probe for specifics about what students did and 
thought in producing a report or project, or might ask 
students to clarify their individual input into a group 
effort. Student projects are often assessed solely 
through the presentation of a finished product: 
behavioral interviewing is a way to make visible the 
often invisible learning in problem-solving or team- 
work gained through projects. 

SHL ASSESSMENT CENTERS 

SHL, an international human resource consulting 
firm founded by industrial psychologists in 1977, _ 
helps organizations assess, select, and develop employ- 
ees at all levels. SHL has developed an array of “assess- 
ment centers.” These are simulations of employee 
tasks and accompanying tools to assess how individu- 
als address work-related challenges, such as function- 
ing in group and team settings, organizing and 
prioritizing critical tasks, and conducting research, 
analysis, and idea generation to solve problems. 
Assessment centers may be job-specific, or they may 
assess for skills across a variety of positions. 



Tools include: 

■ Group exercises with assigned roles, in which 
every participant is given a different set of 
background materials; 

■ Group exercises with unassigned roles, in which 
all participants receive the same background 
materials; 

■ “In-tray” exercises, in which participants must 
analyze and prioritize written documents and 
propose actions; 

■ Fact-finding exercises, through which an indi- 
vidual starts with partial knowledge and must 
decide what extra information is required to 
complete the tasks, often with a “resource per- 
son” available to provide additional information 
as requested; and 

■ Analysis-presentation exercises, in which partici- 
pants must absorb and analyze information, then 
present their conclusions and recommendations 
for action. 

For the group exercises, participants receive materials 
(e.g., site plans, sales figures, staffing requirements) 

and have 1.0 minutes to prepare for the exercise 

individually. The group then has 50 minutes to 
determine a solution to the given problem, such as 
determining the allocation of space in a new office or 
manufacturing space. Trained assessors observe the 
participants’ actions and rate their performance 
according to a scoring rubric. For example, to assess 
“influence,” assessors must determine if tasks are 
delegated appropriately; to assess “team membership,” 
assessors determine if all suggestions of the group are 
considered or dismissed out of hand. 
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The simulations have been used in a wide variety of 
settings for both hiring and staff development pur- 
poses. Because it can be costly to train assessors, the 
simulations tend to be used for senior management 
positions, although companies have used them to 
identify promising college graduates. For example, 
financial institutions have used SHL simulations to 
recruit financial analysts; a major pharmaceutical 
corporation has used them to hire sales staff, and a 
major chemical corporation has used them as a 
developmental tool for its managers around the world. 
A major national insurance company conducted a 
“high fidelity simulation” in which applicants viewed 
a video of the performance of the desired job, partici- 
pated in a tutorial on the fundamentals of the 
company s customer service software, and then were 
assessed as they responded to simulated calls from 
customers asking for assistance. 

Educators interested in experimenting with the 
assessment center process for their classroom might 
adapt a simulation that is not occupation-specific. For 
example, they might assess students’ skills in gathering 
and analyzing information, prioritizing tasks, or 
making presentations. 



SHL North American Headquarters , Flatiron Park West, 
2555 55 th Street, Suite 20 ID, Boulder, CO 80301 
(303) 442-5607; www.shlusa.com 



Competency Assessment in Adult 
Education Settings 

EQUIPPED FOR THE FUTURE 

The National Institute for Literacy, an independent 
federal organization, has developed a set of learning 
standards that are based on the knowledge and skills 
that adults need in order to carry out their roles and 
responsibilities as workers, parents and family members, 
and citizens and community members. Through a 
collaborative and grassroots process involving adult 
education initiatives across the country, the Institute has 
identified skills and knowledge that are critical to family, 
civic, and work life. Equipped for the Future (EFF) 
Standards for Literacy and Adult Lifelong Learning were 
developed through a careful research process with adult 
learners, resulting in a core set of 16 content standards 
that include workplace competencies but encompass the 
complexities of negotiating civic and family life as well. 

The Equipped for the Future skills are as follows: 

■ Communication Skills : Read with understanding, 
convey ideas in writing, speak so others can under- 
stand, listen actively, observe critically; 

■ Decision-Making Skills : Use math to solve problems 
and communicate, solve problems and make 
decisions, plan; 

■ Interpersonal Skills: Cooperate with others, advocate 
and influence, resolve conflict and negotiate, guide 
others; and 

■ Lifelong Learning Skills : Take responsibility for 
learning, reflect and evaluate, learn through re- 
search, and use information and communications 
technology. 
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In developing the standards, collaborators started with 
the question of what adult “competence’’ looks like, 
paying attention to the range of adult learners from 
those with basic skills to those with formal education 
and experience. They have identified four key dimen- 
sions of performance to generate detailed descriptions of 
learner performance. Performance dimensions will 
eventually be used for placing descriptions on a 
developmental performance continuum for each EFF 
standard. These dimensions are: 

■ Depth and breadth of knowledge base; 

■ Degree of independence; 

a Range of conditions under which one can 
perform; and 

■ Degree of fluency and flexibility with which one 
can perform. 

The National Institute for Literacy is using a 
grassroots approach to developing the EFF assessment 
framework. The institute is working with adult 
education programs in five states (Maine, Ohio, 
Oregon, Tennessee, and Washington) to identify a 
research-based performance continuum for each EFF 
standard that will support the identification of level 
descriptors for all 16 standards. Participating teachers 
are collecting information about student performance 
and progress against the content standards, using a 
data collection template that focuses observation and 
documentation of learner performance on the four 
dimensions of skill performance. 

Currently, the most common way that adult education 
practitioners are assessing the EFF standards is 
through the use of portfolios. For example, at the 
Even Start Program in Canton, Ohio, adult learners 
participated in a joint project with a hospital, decorat- 
ing sweatshirts for donation to a local charity. The 



teachers decided to use this activity to teach the skill 
“plan,” one of the decision-making skills. 

First, Even Start teachers reflected on the degree to 

which the students already had the skill of planning; 

they determined what to look for in order to know if 
the students were attaining this skill, focusing on the 
four dimensions of standards, described above — for 
example, can students plan independently? Can they 
plan under a range of conditions? Then, the teachers 
explicitly taught the steps to planning, using the 
standard’s components of performance as a guide. 

With teacher guidance, students developed the plan 
for accomplishing the task of creating the sweatshirts. 
After students carried out the activity of creating the 
sweatshirts, they held discussions to evaluate the 

activity and their planning process. Students then 

completed a written self-evaluation, which they placed 
in their portfolio. 

The assessment of student learning was based on ^ 
teachers’ ongoing observation of student performance 
in this activity and in other arenas of their lives. For 
example, the teachers provided follow-up activities to 
determine if their students had learned how to plan, 
which also gave students the opportunity to transfer - 
skills to another context. In one such activity, students 
had to complete an individual written plan before 
working on creating a “literature kit” for their chil- 
dren, rather than developing a plan as a group with 
teacher guidance. Teachers used this activity as a post- 
test to identify whether students could integrate all the 
skill’s components and in what areas students needed 
further skill development. Teachers also pointed out to 
students when they were successfully integrating the 
skill into their daily lives, such as when they had to 
plan to secure day care arrangements. This helps 
students to reflect on their own skill attainment and 
to convey it to others. 
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Along with other student work, the written plans for 
each activity were included in their portfolios. Stu- 
dents’ accomplishments, documented in the portfo- 
lios, form the basis for “Career Passports” that stu- 
dents use for employment searches. The Career 
Passport summarizes a student’s marketable skills and 
includes a resume, awards or certificates, a list of the 
persons competencies, and letters of recommendation. 

For more information about Equipped for the Future, 
contact the National Institute for Literacy, 1775 1 Street, 
NW, Suite 730, Washington, DC 20006, (202) 233- 
2025; www.nifl.gov. Equipped for the Future publications 
are available from ED Pubs: 1 (877) 433-7827. For 
more information about the Even Start program, contact 
Jane Meyer, Adult Basic and Literacy Education Coordi- 
nator, Canton City Schools, (330) 588-2148, ext. 130. 



Conclusion 

Interest in assessing high-performance skills is coming 
from many quarters: employers, state colleges, and 
universities that reside at the junction between educa- 
tion and the economy; high school educators seeking 
to expand beyond traditional curricula and outside the 
school walls; and parents concerned that their children 
get critical thinking skills necessary for success in 
jDostsecondary education and in high-skill careers. 

This guide can serve as a starting point for those who 
seek to equip young people with a broad range of 
cross-cutting, high-performance skills that will help 
them succeed in postsecondary education and in the 
changing economy. 
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Northeast and Islands Regional Educational Laboratory 

a program of The Education Alliance at Brown University 

The LAB, a program of The Education Alliance at Brown University, is one of ten 
educational laboratories funded by the U.S. Department of Educations Office of 
Educational Research and Improvement. Our goals are to improve teaching and 
learning, advance school improvement, build capacity for reform, and develop 
strategic alliances with key members of the regions education and policy making 
community. 

The LAB develops educational products and services for school administrators, 
policymakers, teachers, and parents in New England, New York, Puerto Rico, and 
the Virgin Islands. Central to our efforts is a commitment to equity and excellence. 

Information about LAB programs and services is available by contacting. 

LAB at Brown University 
The Education Alliance 
222 Richmond Street, Suite 300 
Providence, Rl 02903-4226 

Phone: 800/521-9550 
E-maii: info@lab.brown.edu 
Web: www.lab.brown.edu 
Fax: 401/421-7650 



Jobs for the Future (JFF) is a LAB partner organization. JFF is a national non- 
profit organization that works to strengthen the foundation for economic opportu- 
nity and civic health in America by advancing the understanding of the skills and 
knowledge required for success in the new economy. JFF works locally and nationally 
to develop innovative workforce development solutions that help people make 
effective lifelong transitions between work and learning. 



This publication is based on work supported by the Office of Educational Research and 
Improvement (OERI), U.S. Department of Education, under Contract Number 
RJ96006401. Any opinions, findings, and conclusions or recommendations expressed m 
this material are those of the authors and do not necessarily reflect the views of OERI, the 
U.S. Department of Education, or any other agency of the U.S. Government. 
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